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(57) Abstract 



A method of anonymous targeted insertion of 
indicia into video broadcasts. Individual televisions 
or other video reception devices aie associated with 
set-top boxes that monitor the usage and viewing 
habits of the television set or other video recep- 
tion device. A viewer profile derived from data ac- 
quired from said monitoring is created wherein the 
viewer profile indicates certain characteristics about 
the viewer. This profile is transmitted to a centralized 
database, said centralized database being an interme- 
diate link between tiie origin of the video broadcast 
and the end viewer. TTic purpose of the database is to 
link specific insertable indicia with matching specific 
viewer profiles. The insertable Indicia are encoded 
directly into the broadcast video and re-broadcast to 
the end viewer where the set-top box decodes the 
broadcast video and performs insertion of the indi- 
cia. Thus, the system and method allow advertisers 
to target specific ads or indicia to specific viewing 
profiles. 
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CROSS-REFERENCE TO REUTED APPUCATIONS 

This patent application is related to and daims the benefit of U.S. provisional 
application serial no. 60/034,517 nied on December 20, 1996 entlUed -Set Top Device for 
Targeted Electronic Insertion of Indicia Into Video". 

BACKGROUND OF THE INVENTION 

1. Field of thfi TnvPnHnp 

This Invention relates to set top video reception devices, and particularly, 
enhandng them to allow insertion of targeted indida Into video. 



2. Description of the RelatPri Ar^- 

Electronic devices for inserting images Into live video signals, such as described In 
U.S. Patent 5,264,933 by Rosser, et al. and U.S. Patent 5,491,517 by Kreitman et al., have 
25 been developed and used commerdaily for the purpose of Inserting advertising and other 
indida into video sequences, induding live broadcasts of sporting events. TTiese devices 
are capable of seamlessly and realisOcally Incorporating logos or other indicia Into the 
Original video In real time, even as tiie original scene Is zoomed, panned, or othen/vise 
altered In size or perspective. 

30 

Uve video insertion of indida requires several steps. The event video must be 
recognized, tradced, and adjusted for the potential Insert perspective and occluding 
objects prior to actual insertion. In the systems discussed In U.S. Patent 5,264,933 by 
Rosser, et al. and U.S. Patent 5,491,517 by Kreitman et at. it was assumed that the 
35 broadcaster would perform die complete process, Induding recognition, trading, creating 
an ocduslon mask, warping inserts to correctly matdi the current Image, and correctiy 
mixing the original video, warped Insert and ocduslon mask. 
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In U.S. Patent 5,543,856 of Rosser, et al., a Live Video insertion System (LVIS) 
split into two functional parts is described, with an upstream, "master" part performing 
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recognition and occlusion mask generation, and sending this Information downstream, 
along with various control parameters, to a less computationally endowed downstr^m 
-Slave" part, capable of warping Inserts to correctly mateh the current Image, and correctly 
mixing the original video, warped insert and occlusion masic 

A number of current trends In television, video, and computer technology make it 
feasible and economically likely that a "slave" LVIS unit wfli be included within future set- 
top units. One trend toward b,oadcasters sending compressed video signals directly to 
the home. Compression is driven by a limited availability of broadcast bandwidth, espe- 
dally satellite based broadcasts. In order to dewmpress the compressed vkJeo users 
require a set-top device that has significant computing power and memory. These set top 
devices are required to run decompression algorithms In real «me. Memo.7 and 
computing power could also be utilized to make the set-top device act as a downstream 
"slave" part of an LVIS system. 
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A second trend is the decreasing price of memory and computing power thereby 
increasing a personal computer's ability to process video bandwldths of Information. 

A third trend Is the movement by telephone companies and other wire network 
providers to higher bandwidth networks. There Is also the possibility that the Wortd Wide 
Web, or some similar computer network, could become a means for large-scale data 
exchange or broadcast of high quality video informatfon. Compressed video, still neces- 
sary to traverse nebvorks with limited bandwklths, is decompressed by the personal 
computer receiving the data. The video processing power of the personal computer may 
be sufficient to also be utilized as the downstream "slave" section of an LVIS system. 

Yet another trend Is towards sending digital television signals directly to the 
home, -mis means that the television set Itself will be a digital processor, potentially 
powerful enough to be programmed and used for the image warping and the other 
processes required of a downstream slave unit of an LVIS system. 

In all these scenarios, the significant point, as fer as this invention Is concerned, is 
that the set-top device, the last link of the video transmission chain, has slgniflGint 
computing power and memory. When this computing power and memory Is sufficient for 
the viewer's set-top device to act as the downstream or "slave" section of an LVIS system, 
a very Interesting possibility arises - the possibility to target adverhsing within a masi 
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medium. In particular, It malces narrow casting of advertising possible, particulariy 
insertions, in television and other video transmissions. 

To understand the benefits of narrow casting to television and video audiences, 
5 which is the subject of this application, it is useful to understand the concepts of targeting 
advertising. 

The most pervasive, and precise, of existing methods of narrow-casting or target 
advertising Is direct mail {akB junk-mail) which uses mail to deliver material to selected 
audiences. Ihe starting point for direa mail is a database of addresses. These databases 
can also be aoss-iinked to so called profile factors, or personal infomiation, pertaining to 
the residents at each address. These profile factors are typically age, income, family 
composition, number of children and their ages, type of automobile owned, dwelling type, 
zip code and various other demographic, psychographic and iife-style information. The 
more of these profile factors the data base contains, the more useful it is for targeting the 
advertising. The data base is sorted by computer to generate a mailing list of candidates 
whose profile factors match an advertiser selected sub-set The advertiser believes that 
dients whose profile factors fall within this selected sub set will be more responsive to 
buying the product tiie advertiser is seiling, so tiiat by mailing only to tiiose people, the 
advertiser (or tiieir client) can reach all of tiie audience who are highly predisposed to 
purchase tiieir product, witii tiie minimum of expense. 
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The use of tiiese databases has tiiree problems. The first is tiiat they are only 
effective for mail. The more Influential nnass media, espedaliy television, cannot be 

25 targeted with anytiiing like the same geographic precision because of the broadcast nature 
of the transmission. The second Is the problem of trying to keep the data bases up to 
date. Typical sources of data used to compile such data bases, such as census informa- 
tion, professional licensed databases, credit card transactions, warranty cards, reverse 
directories and consumer surveys can be montiis, and more typically, years out of date, 

30 leading to considerable waste and to missing a substantial fraction of potential prospects. 
Even good data bases only guarantee 80-90% deliverability - i.e. 10 - 20% of the 
addresses are no longer valid. The third is the concern for privacy. The existence of such 
centi3iized data bases worries many people because of their potential misuse by agendes, 
induding but not limited to government agendes, having autiiorized or unauthorized 

35 access to tiie data bases, and also their potential use by criminals for targeting thefl, con 
schemes and other misdeeds. 
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The set-top downstream version of LVIS solves all three of these problems Rret 
■t bnngs the power of direct marketing to video. In particular to the mass marlcet medium 
of television. Moreover, it can do this in a way that avoids the need for centralized data 
bases, with their privacy and out-of-<late concerns. 7T,e proposed targeting mechanism of 
5 this application, Anonymous Target Profiling, effective^ tar^ viewers profile factors 
without making them publicly available In a way that ensures profile factors are close to 
100% current 
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SUMMARY 

TTie invention comprises both a method and an apparatus to act as a Live Video 
insertfon System (LVIS), split Into two functional parts, with an upstream, "master- part 
doing the recognition and ocduslon mask generation, and sending this informaton 
downstream, along with various control parameters, to a less computationally endowed 
downstream "slave" part, capable of warping Inserts to correctly match the current Image 
and correcdy mbdng the original video, warped insert and occlusion mask, where the 
downstream sectton 1$ part of a set- top device In a viewer's home. 

Because of the location of the set-top device at the viewer's television set, it 
becomes possible to narrowest video insertions to a single household, whfch may b^ a 
Single person or even a particular TV set within a household. Narrow casting could be 
implemented as the television or video equivalent of direct mailing, in which a central, 
computer sorted data base is used to select viewers whose profile factors match an 
advertiser selected sub-set. For instance, the geographic location of set-top devices could 
be made extremely local by GPS type devices In the set-top device which may also double 
as theft protection mechanisms, or by phone numbers of attached modems, or postal 
codes, or by mailing addresses which are stored in Hie set top device, possibly as part of 
product warranty submissions. However, the availability of significant memory and 
computing power in the set-top device opens up a much more exciting possibility, which 
we term Anonymous Target Profiling (ATP). 

Anonymous Target Profiling does not require a centralized database of all potential 
clients. Instead, there Is a viewer usage recorder or monitor, located at the viewer 
location, and a viewer usage interpreter or key, supplied witti ttie broadcast. The viewer 
usage recorder or monitor is a system which monitors television usage patterns and stores 
a continuously updated version of a usage profile. TTie set-top device is an ideal place to 
locate a viewer usage monitor. In a simple fom. ttie viewer usage monitor would classify 
programs (or channels) and record a rolling viewing profile of viewing habits, including 
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type of program watched, time of day and day of the week of viewing the program and 
duration of that viewing. More complex models of viewer usage may also indude programs 
not watched, Intensity of viewing (I.e. volume adjustments), surfing patterns (i.e. what 
video snippets arrest the attention of a channel surfer, even for a short time) and other 
5 more subtle aspects of viewer interaction with the medium. 

A viewer usage Interpreter is a key that translates the viewer usage profile Into a 
set of profile factors associated with the viewir>g pattern. The viewer usage interpreter 
could be generated statistically by having a sample of households of known profile factors, 

10 who have their viewing habits monitored by a central system. By choosing the sample 
households scientifically so that each household in the television viewing population has a 
known chance of selectton, the results obtained from the sample can be reliably projected 
to a larger television viewing audience. The sample size required for survey depends on 
the reliability needed. A moderate sample size is sufficient for most needs. For example, 

15 national polls, such as those conducted by the weit known Gallup or Harris organizations, 
generally use samples of about 1,500 persons to reflect national attitudes and opinions to 
within an accuracy of ± 4%. A sample of this size produces accurate estimates even for a 
country as large as the United States with a populatksn of over 250 million people. 

20 In one usage of the invention, a broadcaster would establish a continuous survey 

of a few thousand households of known profile fadors for each significant broadcast 
region. These surveys would be used to generate cross-correlations between viewer usage 
profiles and viewer profile factors. Advertisers wishing to have their advertising targeted to 
viewers with a particular sub-set of profile factors would be able to use the cross- 

25 correlations to translate tiieir viewer profile requests Into a viewer usage profiles request. 
The broadcaster would tiien send tiie required viewer usage profiles as part of the 
broadcast in for instance, the vertical blanking interval (VBI) along witii tiie advertisers 
insertion also in ttie VBI, over a number of fields, if necessary. At tiie viewer's set-top, tiie 
device would see which insertion was linked to tiie local viewer usage profile, and insert 

30 appropriately. 

For instance, on a widely watched event, such as the super bowl, a car company 
may chose to present different models, depending on the demographic or psychographic 
profile of tiie family, based on tiieir viewing habits. As a simple example, a family witii a 
35 viewing profile that Includes significant viewing of young children's programs Is assumed 
to have children and may be shown advertisements for a mini-van, while a family witii a 
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profile that indudes signifiont viewing of programs for out-door sports may be shown an 
advertisement for a sports utility vehicle made by the same company. 

There could also be a Vrite-in" dimension to the viewing providing the viewer the 
5 opportunity to select extra specific profile factor?. For instance, viewers who are looking 
for a car may add this fact to their viewer profile in order to deliberately solicit 
advertisements for cars. It may also be possible to spediy price ranges and other relevant 
parameters. 

10 In a further embodiment of the Invention, insertions could appear as border 

advertisements surrounding, or partiy surrounding, one or more of the video windows, or 
could be a separate video window, which may change position, size, shape and orientation 
on the screen as a means of increasing tiie impact of the advertisement 

15 In anotiier furttier embodiment of the invention, the set-top device could be used 

not only for in-programming advertising, as made possible by the LVIS system, but could 
use one or more secondary, possibly compressed, video channels as a source of alternate 
advertisements for showing in the conventional advertising breaks. These inter-program 
advertisements, usually coupled to events in the program, could also be at suitable breaks 
coupled to viewer action, such as when a viewer first turns the set on, switches channels, 
turns the set off, or alters viewing parameters, such as volume. 
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In a still further embodiment, in which the television is also connected to a 
computer networic, such as the Worid Wide Web, the viewing profile could be extended to 
25 indude a browsing profile, related to frequently visited web sites or other services 
requested. In addition, the advertising inserted could be web site addresses or other 
fonns of links to furttier information, or furttier advertising, related to the product being 
advertised. 
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Much of tiie technology needed to Implement ttie viewer usage monitor, necessary 
for Anonymous Target Profiling, could also be used to provide "smart" TV sets, which 
would favorably impact the economics of implementing the invention by allowing tiie set- 
top manufacturers or distributors to offset a substantial part of the cost of the set-top 
device to tiie end user. For example, a smart: TV set, when turned on, would not just be 
on the channel it was on when It was turned off. Depending on the time and day of the 
week, it would turn on to the channel indicated most likely by the viewer usage profile, 
regardless of where it was when it was turned off. A smart: television may also be used to 
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provide user customized bum-ins, espedaliy ones similar to those used b/ broadcasters to 
show baseball and football statistics. The exb^ channete and turners necessary for tiie 
network to offer alternate, full video to its advertisers, could also be used to have multiple 
windows, i.e., enhanced picture in a picture. Multiple windows would also enable ttie 
5 ability to turn on witii predetermined setups more compelling. The warping necessary for 
the downstream, slave LVIS system, could be used to make one or moi^ of these windows 
re-sizable, magnifj^ble (for people who wanted to examine some detail of the video) and 
even rotatable (for people who may want to lie down and have tiie video on its side as 
well). Writable digital video disks, or otiier high capadty, random access memory could be 
used by advertisers to store full motion video for insertion at tiie appropriate time. Such 
devices can also provide viewers with tiieir own instant replay feature, automaticaliy 
storing tiie last five or more minutes of what ever program was being watched. This 
feature would also make the magnification capability more compelling, especially for 
example to sports fans who rriay wish to go back and look at some aspea of play such as 
a ball landing ctose to a line in detail. Writable devices can also art as a scrap pad for 
grabbing bits of video tiiey want to see later or show someone else; or as a more 
conventional video recorder. 
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These additional features may also be used as ti-igger^ for showing live or still 
video advertisements, eitiier before or after the feature is used, or as a bortler 
advertisement during the use of tfie feature, or as a live video insertion on some 
recognized part of the video. 

In addition to processing insertion information from a pattern recognition version 
of an LVIS system, the downstream, set-top part of tiie LVIS system could be taking 
insertion information from camera head sensors, including the types of systems developed 
for virtual studio systems, or it could be taking Information from multi-user game 
applications. 



BRIEF DESCRIPTION OF THE DRAWINGS 

Rg. 1 is a schematic diagram showing an LVIS system split into an upsti^am 
"master" and downstream "slave system". 

Rg. 2 is a schematic diagram, showing details of ttie end-user set-top device, 
enhanced to enable it to perform as tiie downstream, slave part of an LVIS system. 
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Fig. 3 shows an exampte of a viewer usage profile, shown as a bar chart. 

Fig. 4 is a schematic diagram, showing details of an alternative embodiment the 
end-user set-top device, enhanced to enable it to perform as the downstream, slave part 
of an LVIS system and to act as a smart TV. 

Rg. 5 is a table showing t/pes of television programming and the percentage of 
total air-time within each type in one US location in a given week In 1986. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

During the course of this description like numbers will be used to Identify like 
1 5 elements according to the different figures which illustrate the invention. 

In the preferred embodiment of the present invention, a video transmission, which may, 
for example, be a live television broadcast of an event being played on a court 10, is 
captured for remote viewing by television cameras 12, and is composed Into a program'for 
viewing within a standard video production unit 14, which may be a television productton 
truck or a video studio, equipped with well known video production equipment. After 
being composed into a program, the video is fed through the front end of a Live Video 
Insertion System (LVIS) 16. This front end of LVIS 16 perfomis the initial functions of 
recognition using the recognition unit 18, tracking using the tracking unit 20 and occlusion 
25 mask production using the occlusion mask production unit 22, as discussed in detail in U.S. 
Patents 5,264,933 and 5,543,856, as well as co-pending patent applications: Serial No, 
08/563,598 filed November 28, 1995 entitled '^SYSTTCM AND METHOD FOR INSERTING 
STATIC AND DYNAMIC IMAGES INTO A LIVE VIDEO BROADCAST; Serial No. 08/580,892 
filed December 29, 1995 entitled "METHOD OF TRACKING SCENE MOTION FOR LIVE 
30 VIDEO INSERTION SYSTEMS"; Serial No. 08/652,089 filed June 12, 1996 entitied "SYSTEM 
AND METHOD OF REAL-TIME INSERTIONS INTO VIDEO USING ADAPTIVE OCaUSION 
WTTH A SYNTHEHC COMMON REFERENCE IMAGE- and Serial No. 60/031,883 filed 
November 27, 1996 entitled ^^CAMERA TIIACKING USING PERSISTANT, SELECTED, IMAGE 
TEXTURE TEMPLATES", the teachings of which are hereby Included by reference. 



35 



The recognition and tracking parameters may also be provided by sensors 13 
attached to the camera Itself, and interpreted by a camera head data Interpreter 15, as 
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used by some virtual reality studio systems, and as discussed in detail in U.S. Provisional 
Application Serial No. 60/038,143 filed on November 27, 1996 entitled ^IMAGE INSERTION 
IN VIDEO STREAMS USING . A COMBINATION OF PHYSICAL SENSORS AND PATTERN 
RECOGNmON"; the teachings of which are hereby included by reference. 

5 

However, unlike the systems discussed in U.S. Patents 5,264,933, the front end 
LVIS system 16 does not use this information to merge the insertion with the live video. 
Instead an encoding unit 24 inserts the information obtained by die other parts of the 
LVIS front end 16 Into the vertical blanking interval of the video or other appropriate co- 
signal such as, but not limited to, a spare audio channel. In addition, encoding unit 24 
may also insert ail or any of a graphic or video for insertion 26, a program categor/ code 
27 or one or more user profile and enabling keys 28. This may be done over any number 
of video fields: a user enabling key (as discussed in detail of U.S. Patent 5,543,856,) and 
one or more viewer usage profile keys 120. The output of encoder unit 24 may be a 
standard video signal which may be the well known standard tOSC or PAL television 
signals with extra information encoded in, for instance, the vertical blanking interval or an 
otherwise unused audio channel, or it may be such a compressed video signal. 

The signal produced by the LVIS front-end 16 is sent via appropriate means 30, 
20 which may be a satellite uplink, or telephone company lines, etc., to a central studio site 
34 for possible further processing before being rebroadcast to a wider audience, which 
may be the general public. The central facility 34 may be responsible for inserting any or 
all of a graphic or video for later insertion by the downstream part of the LVIS system 46, 
a user enabling key, one or more viewer usage profile keys, and a prxjgram category code, 
25 all for use by the downstream part of the LVIS system 46. The central studio site 34 would 
also be responsible for supplying conventional video advertising which may also be 
targeted using the Anonymous Profile Targeting methodology of this application. After 
appropriate alterations are nr>ade by the central studio site 34, the signal is distributed via 
suitable distribution means 40 and 42 which may be a satellite transmission system, a 
30 cable network, a terrestrial broadcast system, computer networic or other appropriate 
means of transferring video or television signals to the end user. 
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The end user has an appropriate reception device 42, which may be a cable 
connection, a conventional TV aerial, a satellite dish, a telephone company line or other 
35 appropriate means of receiving television or video signals. After reception, the signal Is fed 
to a set-top device 44 before reaching the end user's video display screen 55 which may 
be a television screen, a computer monitor or otiner appropriate display medium. The set- 
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top user may have appropriate means for decompressing 52 signals as well as other 
suitable control devices 54 which may perfomi various functions that make the set-top 
device desirable tx, the end user, such as but not limited to, custom^ed bum-ins, 
automate channel selection on power up and magnification of or re-sizing of extra viewing 
5 windows. The set-top device 44 of the preferred embodiment has, as a minimum, the 
components of a downstream LVIS system 46, with the ability to strip-off, interpret and 
use the Information mixed in with the video signal by the up-stream LVIS system 16. In 
particular, the down str^m unit 46 is able to use the infomiation generated by the 
recognition unit 18, the tradcing unit 20, and the ocdusion mask production unit 22 to 
perfofTD seamless insertion of still, animated, and live video Indicia into the video stream in 
a way that can make the inserted indicia appear to the end user as if it were part of the 
original scene 10, 
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•nie set-top device 44 of the preferred embodiment is also capable of stripping off, 
interpreting and using any of a graphic or video, a user enabling key, one or more viewer 
usage profile keys 120, and a program category code, each of which may have been 
attached to the video stream by the encoding unit 24 or by central studio site facility 34. 
In particular, by comparing the viewer usage profile keys 120 with the local viewer usage 
profile 50, different insertions 58 and 60 may be made on different end users video 
viewing devices 56. The different insertions may be permanentiy stored locally in memory 
device 55, or downloaded there during or prior to transmission of the live video 
transmission in which they are inserted. 

TTie end-user set-top 44 of ttie preferred embodiment is shown in greater detail in 
25 the schematic drawing of Fig. 2. The input data stream 70, which may be broadcast video 
or another suitable means of transmitting video to an end user, inciuding but not limited to 
analogue or digital television broadcast, or MPEG2 or otiier compressed video, is typically 
received via a selection device 72, which may be, but is not limited to, a standard 
television tuner. The function of tiie selection device 72 is to discriminate between the 
30 variety of different video programs or data streams which may be being distributed over 
the same channel but on different frequency bands, or deriving from different locations on 
a networic. In the preferred embodiment the selection device 72 Is monitored by a viewer 
usage profile generator 74, which also has access to the current time and date via a dock 
76 and to tiie type of program from the type of program indicator 78, which has been 
35 given this information by tiie vertical blanking interval decoder 80. The function of the of 
the usage profile generator 74 is to build up a history of the use of video viewing. The 
pattern of viewing, which may include type of program, nature of program being watched, 
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time of day watching the program, day of weeic of watching the program, duration of 
watching of the program and any other relevant Information pertaining to the program 
can be used to predict, within acceptable margins of enor, the so called profile factor^ 
which direct market advertisers currently obtain from demographic and psycho-gi^phic 
5 data bases. 

Tlie profile factors could be determined to different degrees of accuracy from 
different amounts of data. For Instance, a viewing profile whfch on^ took into account 
dme of day, day of week, time of viewing, duration of dewing, Including channel surfing 
patterns - i.e. all data that requires no program or channel labeling - would reveal a great 
deal about the viewer but would leave substantial room for error. For instance, a pattern 
that had viewing on weekdays from 6:30 Al^ to 8:00 AM but nothing again until 7:00 PM 
to 11:00 PM, and nothing on Saturdays until after 10:00 AM, would indicate a house hold 
with no children, all members of which work. White this Is a useful conduston, and may be 
used for a degree of targeting, it does not necessarily give any indication of, for instance, 
the actual size of the viewing family, or the gender make-up of the family. By including 
monitoring of which program types were viewed, considerably more analysis is possible, 
with a good probability of being able to predict such important profile factors as gender! 
age and income. 

In the preferred embodiment, the interpretation of the viewer usage profiles i.e. 
the cross - correlation between viewer usage profile and viewer profile factors which we 
have temied the viewer usage profile key would be established using well known suivey 
sampling techniques, and practiced by such companies as the well known Gallup or Harris 
organizations. The viewer usage profile key couid be generated by having a sample of 
25 house holds, of known profile factors, who have their viewing habits automatically 
monitored by a central system, which may be a computer linked into tiie viewers set top 
device by a modem and telephone link, or other appropriate technology. By choosing the 
sample households scientifically so that each household in the television or video viewing 
population has a known chance of selection, the results obtained from the sampling can be 
, 30 reliably projected the television or video viewing public 

The accuracy and significance of the viewer usage profile key will depend on the 
data used, and how it is used, to obtain viewer usage profile 120. A diagrammatic view of 
a viewer usage profile is shown as a bar chart in Fig. 3. Horizontal axis 122 is used to 
35 represent program category, and vertical axis 126 is used to represent a measure of 
viewing intensity associated with each program categor/. A simple form of that measure 
is duration of viewing of a program category. A typical entry can be represented as a bar 
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124. In a simple embodiment the program categories would be Hme of day and day of 
week. For example, the first category (i, i) may be a program shown between 12:00 
midnight and 1:00 AM on Monday. The total number of categories in such a scheme could 
be 168, Le., the number of hours in a week. Vertical axis 125 in this scheme would 
5 represent time spent viewing any particular category, shown graphically by the height of 
corresponding bar 124. In general, the program category would be a vector string each 
element of the sWng corresponding to an attribute. Attributes could Include time of day 
day of week, month of year, day of year, type of program, channel of viewing, broadcasted 
of the Video being viewed, sponsors of material being viewed, whether the program being 
watched was a rerun, when the program was made, where the program was made, 
producer of the program, major actors in the program, director of the program, or any 
other relevant attribute. The program category vector generally consists of three 
attributes: the day of week, time of day, and program type. 
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This necessitates a coded fam, of the type of program being transmitted by the 
broadcasters, preferably in the vertical blanking interval, though it may be on or encoded 
in a spare audio channel, or In the video itself, either in some spare or extra fieWs In for 
Instance, the tfUe, or opening sequence, or In the credits. The program type may be 
encrypted in the vWeo itself either as a bum in or some alteration to a bum in. In a digital 
btoadcast it may be enoypted as a least significant bit pattern, as is becoming common in 
digital image authentication schemes. Typical program types might Include specific sports, 
(football, baseball, basketball, ets.) each of which may have sub-categories, such as major 
league, minor league, news, current affairs, film (with sub-categories). 

Richard F. Tafllnger, Professor at the Edward R. Murrow School of Communica- 
tion, Washington State University in "Sitcom: What It Is, How It Worics", lists twenty-six 
different primary types of television shows. Rg. 5 shows a list of both these twenty-six 
types and the percentage of each which was available in a given week's television 
transmission in 1986 from an area covered by seven program providers. Professor 
Tafllnger also notes that the average family in 1992 watched television seven hours and 
seventeen minutes a day, or over 50 hours a week, more than the average woric week. 
Televisksn is obviously a major component of American life, and because of botii the 
diversity of viewing available, and the Hme spent viewing, patterns of TV viewing can be a 
very powerful tool for determining both demographic and psychographic make up of the 
viewing family. Program types could be sub-divided. One major subdiviston could be 
whether a program vras a re-mn or not, and if a re-run, how recent. 
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Vertical axis 126 of the example viewer usage profile 120 would record the 
viewing intensity experience associated with each of the program categories represented 
by horizontal axis 122. A simple measure of viewing Intensity is time spent viewing that 
particular program category. This may be represented as total accumulated time, the total 
5 time in the last month, or some rolling time average. Additional factors, such as but not 
limited to, volume, increases in volume, whether or not the set was already tuned to the 
channel, whether the viewer joined the program late, or whether the viewer left the 
program early, could be used weight the time watched to more satisfactorily generate a 
compound merit function or estimate of the intensity of the viewing experience associated 
with each of the program categories. In the preferred embodiment, viewing intensity 
associated with each of the program categories is simply the un-wefghted, roiling average 
of time per weei; spent watching that category, with a weighting function that gives the 
current week unit weighting, and tiien systematically reduces the weighting of previous 
weeks. One simple way to do mis is to add tiie previous average of each program 
category to ttie cun-ent weekly total for each program category and divide by two. Many 
other algorithms could be devised to achieve a similar result 
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In the prefen-ed embodiment, after leaving tuner 72, Uie signal goes to the de- 
compressor, which, if necessary does any decompressing such as, but not limited to, well 
known MPEG2 decompression. The output Is a base-band video signal 84, which is split 
into two, one copy of tiie signal going to a delay line 86, and the other part to the vertical 
blanking interval decoder 80. The function of vertical blanking interval decoder 80 Is to 
extract tiie information that was placed there upstream by eitiier LVIS front-end 16 or by 
central studio site 34. In particular, vertical blanking interval decoder 80 extracts model 
25 information 88, occlusion mask 87, the images or videos to be inserted 90, any auxiliary 
text information 92 associated witii tiie insertion, the required viewer profiles 94 
associated with the different insertion videos 90, and different texts 92. 



30 



For Instance, one use may be to have a single video insertion 90 of a product, but 
wiUi a number of different texts 92. The default text may be in English, but for viewer 
usage profiles 74 tiiat show usage of particular ethnic channels, such as Spanish language 
channels, the text may be In Spanish. Matching a viewer usage profile 74 of tiie current 
set-top device 44 and the required viewer usage profile 94 is done by profile matcher 96 
which selects required text data 92 to be fed to text-to-video converters 98. Profile 
35 matcher 96 also selects which of the stored video insertions 90 are fed to warp unit 100. 
Warp unit 100 takes tiie appropriate model information 88 and uses it to warp tiie 
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appropriate text video 98 and the appropriate video insertion 90 into the appropriate pose 
required to make the insertion behave as If it were part of the natural scene. 

Ocdusion mask 87 is also fed to a warper 89 which uses model information to 
5 warp the ocdusion mask into the appropriate pose for the final video. Mixing unit 102 
then combines the warped occlusion mask, the warped insertion video and text-video with 
base-band video 84 which has been delayed by delay line 86 for the time taken to decode 
and warp the Images Into place. The composite output of mixing device 102, which is a 
Video signal with an Insertion In place is fed into a channel modulator 104, which converts 
the base-band video to die form expected by the selected channel of a standard NTCC 
telev«,on set, as Is customary. Obviously similar arrangements could be made for other 
televBion fbmiats, such as but not limited to the well known PAL, SECAM, digital and 
HDTTV, and other channels of the television receiver. TTw resultant signal Is then sent to 
the end user's television set 105 for viewing by the end user. 
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An alternative, more generalized version of the set top device Is shown 
schemaucally in Hg. 4. The input data stream 70 can now be one of a number of 
communication channels, including but not limited to, a telephone/Internet connection 
130, a cable video connection 132, a broadcast video aerial 134 and a satellite dish 136. 
Each of these data streams Is selected by the appropriate selection device, including but 
not limited to, a modem 138, a cable modem 140, a television tuner 142 and a satellite 
decoder 144. Each of the selection devices is controlled via a central conboller 146, which 
may be a prt)grammable microprocessor, whfch is in turn controlled by the user, by some 
device such as a conventional television r^ote control 71, or some modified version 
thereof, via the viewer control interface 148. From the selection devices, the incoming 
signal goes either to a video and audio router 150 or a data router 152, both of whfch are 
under the control of central controller 146. The video and audio router 150 is linked to a 
video and audio storage device 152, which may be a well known electronic RAM memory 
or a well known device, such as but not limited to a writable Digital Video Disk (DVD), and 
a number of video and/or audio processing devices perfbmiing well known funcdons or 
operations, Including, but not limited to, a decompression device 154, a video and audio 
mixing devfce 1S6, an ocdusfon mask generator 158, an video or image warper 150, and a 
channel modulator 162. Each of these devices is under the control of the central controller, 
as Is the video storage device 152. The video router ISO Is also linked to a video and audio 
interpreter 164, which is capable of extracting data embedded in, or attached to the video 
or audio channels, as for instance data included in the vertical blanking inter/al. The data 
router 152 has connections to a text-to-video and/or audio converter 166, which can be 
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fed on to the video router 150, the interpreter 164, and a data storage device 158, which 
Is also linked to the Interpreter 164 and Is under the control of the central controller 146. 
The central controller also has connection to the viewer usage profile store 170 and to a 
location information store 172 and a dock 174, which Is capable of supplying time and 
5 date information. The location information store may Indude a well known Global Position 
Satellite (GPS) sensing and interpreting device. 
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In normal operation the viewer InterBCts with their television set via the remote 
control device 71, or other similar viewer controlled device sudi as but not limited to 
buttons or switdies on the viewer set or set top device. The viewer operations of turning 
the set on or off; channel selection; adjustment of parameters induding, but not limited 
to, volume, brightness, contrast etc., and other viewer usage diofces are handled by the 
viewer control interface 148, whldi may be a graphic user interface displayed on the 
viewers television or video display. Ttie viewer requests are passed on to the central 
15 controller 146, whidi is typically a programmed micro-processor, as is well known in the 
art of embedded control technology. 

One of the functions of the central controller is to carry out the viewer instructions 
by setting up the appropriate connections between all the appropriate modules within the 
set top device. This Indudes selection of the modem, cable modem, tuner or decoder as 
the primary receiving device; setting up of that primary receiving device to the appropriate 
channel, bandwidth or address to receive the data or program requested by the user; and 
using the data and video routers to direct the television video, audio and data signals via 
the appropriate storage and processing devices, induding but not limited to the video and 
25 audio storage unit 152, the decompression unit 154, the video and audio mixer 156 the 
occlusion mask generator 158, the warper 160 and the diannei modulator 162, so that 
the viewer ends up with the information requested, which may be a television program, or 
a text or image page in hyper text mark up language (KTML) or virtual reality modeling 
language (VRML) or other suitable protocol, from the worid wide web, or some 
30 combination of sudi sources, displayed in the appropriate form on their end viewing 
device 106, whidi may be a television set or a computer monitor or other suitable means 
of displaying video or television information. 

In addition, the central controller 146 is monitoring the viewer's choices by 
35 monitoring the download or interpretation devices whether they be a modem 138, a cable 
modem 140, a television turner 142 or a satellite decoder 144 and using the settings or 
mode of those devices, along with data from the dodc 174, location infonnation unit 172, 
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viewer control interface 148, and informatfon gleaned from interpreter 164, to build up a 
viewer usage profile 120 and store this in the viewer usage profile store 170. Viewer 
usage profile 120 stored in viewer usage profile store 170 Is a measure of the temporal 
pattern and viewing Intensity associated with each of the program categories available to 
5 the viewer and du;cemibie by central controller 146. Typically the program categories 
would consist of a list or vector of atfributes, where the attributes may include, but are not 
limited to time of day, day of week, month of year, day of year, type of program, channel 
of viewing, broadcaster of the video being viewed, sponsors of material being viewed, 
whether the program being watched was a rerun, when the program was made, where 
the program was made, producer of the program, major actors in the program, director of 
the program, or any other relevant information. 
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Each of these attributes may itself have a number of divisions and sub^Jivlsions. 
For instance, the important attnbute of type of television or video program may include, 
but is not limited to, such types as movies, situation comedies, cartoons, news, drama, 
soap operas, sport, children's shows, game shows, religious show, crime shows, music 
shows, talk shows, information shows, comedy, infomercials, entertainment shows, action 
shows, science fiction, shopping services, health shows, mystery shows, western shows 
and education shows. Sport may be subdh^ided for Instance, by broad categories such as 
live or recorded, by continent or country of origin, or by type of sport such as, but not 
limited to baseball, football, hockey, basketball, tennis, soccer, rugby, cricket, bowling, 
track and field etc. Even the type of sport may be subdivided by such categories as 
amateur or professional, international or local, pre-season, regular season, or post-season, 
major league or minor league. 

Similarly, most of the types of television or video programs could be subdivided by 
broad or nanxjw classes along lines appropriate to the specific genre of program. 
Although it may be possible for automated pattern recognition to recognize some of these 
types of programs it is assumed that most of the information regarding program type will 
be encrypted in or attached to the broadcast or transmission of the video or television 
signal upstream of the set top device. 

One method would be to include a standardized code for the type of program, an 
appropriate part of the vertical blanking interval, or as a suitably coded part of a station 
identification bum in. Interpreter 164 may strip the appropriate program type infomiation 
out of tiie video or television signal send it to tiie central controller for use in building up 
the local viewer usage profile. 
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user's viewing device 106. The appropriate advertisement on any given set top device 
would be the one where the tocal viewer usage profile matches or falls within the 
parameters of the required viewer usage profile attached to the advertisement. A default 
advertisement may be shown to homes where the viewer profile does not match or fall 
5 w,thin the profiles or profile ranges requested by the advertisers targeting their 
advertisements to specific audiences. The required viewer usage profiles requested by the 
advertiser, and attached in some fom, to particular advertisements and insertions, may 
take the fbm, of ranges of viewing intensity of one or more program categories or groups 
of categories, or it may be sent as a required range of ratios of the viewing intensity of 
one program category witii respea to one or more other categories. 

For instance, an advertiser may want their advertisements for a particular baby 
product shown only to house-holds which have a certain average viewing of both day time 
soap and day time children's shows but no significant viewing of sports. The reason for 
doing this may be because tiiey believe that those parameters are an accurate way to 
target tiie families that are most likely to buy or use their products or services which may 
be, for instance, single parent mothers who have children and use live-in day care. 
Naturally, the same surveys used to compile the cross correlation keys would also be 
useful in estimating, to the same degree of accuracy, the size of the viewing population 
ha>rfng viewer usage profiles that fit the advertisers requirements, and hence how much 
the advertisers should pay for their advertisements or insertions. 
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An alternate way of expressing the required viewer usage profile may be to show 
the partricuiar advertisement or insertion only to households where the ratios of the 
25 viewing intensity of some particular program categories, or groups of categories, exceed 
some minimum, fall beneath some maximum or lie within some range of values. For 
instance, tiie requirement may be to only show the advertisements or insertions to the 
households where the ratio of average time spent viewing major league baseball to the 
time spent viewing situation comedies was one or greater. Or, the advertisers requirement 
may be that only households where the time spent watching day-time game shows was 
roughly equal to the time spent watching home shopping programs. The reason for 
requesting restriction to those house-holds would be because in the survey sample 
population these ratios had been found to be reliable indicators of some demographic or 
psychographic attribute of the viewer or viewing family that defined the advertiser's or 
odier system user's desired audience. Program producers or broadcasters may ttiemselves 
wish to use the system of Anonymous Profile targeting to present different versions of 
programs to different households. For Instance, house-hoids whose viewer usage profile 
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Another method may be to make use of the program codes supplied by many 
broadcasters that allow easy programming of television sets, or simply to provide the 
information about what settings get what channel to the central controller. 

In addition, In a statistically representative sample of the viewing population, 
central controlier 146 would send the same Infomiatlon being used to compile the local 
viewer usage profile 170 back via the data router and one or other of the 
internet/telephone modem or the cable modem, back to a central collection point. At that 
central site, the data would be correlated and compared with databases containing 
demographic and psychographic information about the same statistical sample of the 
viewing population. Significant con^atlons between the viewer usage profiles and 
Important profile factors, such as but not limited to, age, gender, income, ethnic origin, life 
style (e.g. married, single), taste, spending habits, credit card usage, employment status, 
15 risk taking profile, education etc. could be extracted. These correlations would provide a 
key for advertisers wishing to target families or individuals with particular profile factors. A 
significant advantage of using these correlations would be on popular shows, wher^ a 
large cross-section of the population, was viewing. The viewing audience couid be 
segmented by profile factors allowing the broadcaster to sell different segments of the 
audience to different advertisers, even within regions covered by a single broadcast 
transmission device. This capabifity will be particularly useful to satellite providers where 
the transmission typically has a large geographical footprint. 

One method of achieving this market segmentation by profile factor. Is the 
25 following. While set-top central controller 146 is routing the viewer requested video, 
television or other source to the end user's set, it may also be routing alternate video or 
television feeds, either by different channels in the same down loader, or by different 
down-loader to video and audio storage unit 152 or data store 165. This alternate video 
feed would typically be relaying a number of different advertisements with a requested 
viewer usage profile or range of profiles suitably associated, attached to or encoded in, 
each particular advertising sequence. The contents of the alternative feed may be stored 
in video and audio storage unit 152. At the appropriate time and place for advertising 
insertion, which may be, but is not limited to, a conventional advertising break, or when 
the viewer changes channel, or when a particular image or scene is in view, the central 
35 controller will use video and audio router 150 and data router 152 and whichever is 
necessary of the other video and audio function modules, including but not limited to, 
video and audio storage device 152, to place an appropriate advertisement on the end 
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indkated the presence of young children may be shown alternative versions of certain 
program in which either violent or sexually explicit scenes were either omitted, or had less 
Violent or explldt scenes substituted. It may be that at different times of the year the 
viewer usage profiles required to reach particular demographic or psychographic attributes 
5 change. 
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Another function of central controller 146 may be to make the set-top device act 
as a downstream or "slave" section of an LVIS system. In partiojlar central controller 146 
would use the set-top device's 44 resources to strip off, interpret and use the information 
attached to, or encoded in, the video or television signal by some up stream LVIS system 
16. In particular, Interpreter 164 would obtain information put in the video, television or 
data stream by recognition unit 18, tracking unit 20, occlusion mask production unit 22, 
and camera data Interpreter 15 of the front-end or upstr^m LVIS "system 16. The same 
sort of information provided by the front-end or upstream LVIS system 16 may also have 
been put In the video, television, or data stream by computer, as for instance, but not 
limited to, part of some video game, particularly multi-user video applications, or as part 
of a well known virtual studio set up. The LVIS information extracted by interpreter 164, 
may be temporarily stored in e data store 168 for use at a later, appropriate time, or used 
immediately to extract appropriate material from data store 168, the video and audio 
storage unit 152, which may be video and audio insertions, and direct it via the 
appropriate additional desk top functional units. 



In addition, central controller 146 will use information about the required viewer 
usage profile attached to each proposed insert, and have compared it with viewer usage 
25 profile 120, stored in viewer usage profile store 170 to decide which insertion to use. The 
selected insertion is, if necessary, decompressed using video and audio decompression 
unit 154, before being warped to the appropriate pose by video warping unit 160. Warper 
160 Is fed appropriate parameters via central controller 146, which has obtained them via 
interpreter 164. After warping, the insertion is mixed into the video and audio stream 
being sent to the end user's set 106 via video and audio mixer 156. This mixing also 
Indudes an occlusion mask, which has been generated by ocdusion mask generator 158, 
which has also received parameters supplied upstream via video and audio interpreter 
164, and has also been warped Into the correct pose via warper 160 using parameters 
supplied upstream by interpreter 164. The result is that the viewer sees an insertion on 
35 the viewing screen which appears to be part of the original video television, data 
broadcast, or transmission, but which has been put there by the combined action of the 
original upstream LVIS, or other placing mechanism, and the set-top devk:e, and is further 
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dependent on the local viewer usage p^fiie or hbtcy of the viewer's usage of that 
particular set 

5 store 17^?^ T"^""^ P^^'e 
5 ^re 170 may be related to a viewer acress tey, or other form of ident,«catlon, so that 
the viewer usage profile relates to a specific individual. 

InastillfurtherembodimentoftheinvenOon, viewer usage profile sto,e 170 may 
^ o^-h.; or ,n part, tocted ins^e the viewers remote a,ntro. 71, or o««r related dej 
that the vtewer uses to acxess and contn,! the content aching the end user display or 
te^K,n set. 7.. related cont«, device, indudes, but is not limited to, hand L 
compute,,, personal «,mputers, Joy sticks, web browsers, and other similar hartware or 
sofhvare modules that may be used to control the date. In add«on, the view or usage 
pr^fite 120 Stored in the viewer usage profile is linked by acress number, or other suitable 
«lentification means, to either an individual person or Individual device, or module, so that 
profiles may be constructed and stored for those indhriduais or Individual devk:e or 
software module. In addltton, ar, IndMdual device or software module may construct and 
store Viewer usage profiles 120 for a number of different individuals, who may be 
dentlfied by name, password, number, or other suitable identification means, including 
but not limited to biometric means such as signature, fingerprint, or retina pattern. 

It is to be understood that the apparatus and method of operation taught herein 
are .llustrative of the invention. Modifications may readily be devised by those skilled In the 
art without departing from the spirit or scope of the invention. 
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WHAT IS CLMMmry;; 

1. A set-top device for sending and receiving data pertaining to television viewing in 
whidi selected video indida or sequences can be viewed comprising: 

means for monitoring the usage of a television; 
5 means for creating a viewer profile based upon the data acquired by said 

monitoring means; 

means for locally storing said viewer profile in said device; 
means for using said localty stored viewer profile to determine which insertable 
video indicia or sequence to insert based upon the required viewer profile encoded in or 
10 along with the video sequence; 

means for receh^ing a video signal, said video signal encoded with Insertable video 
Indicia and data pertaining to which Indicia or sequence to insert and where and when to 
insert said indicia or sequence; 

means for decoding said received video signal; and 
15 means for inserting said indicia directly into said video signal for viewing on said 

television. 

2. Tlie set-top device of daim 1 further comprising: 

means for sampling a statistically representative sub-set of viewers to create 
statistical correlations between viewer demographics and the set-top user's viewer profile. 
20 3. -me set-top device of claim 2 wherein said monitoring means monitors time of day, 
day of week, duration of viewing time per channel, channel tuned, prxDgram type being 
viewed, and channel surfing pattems. 

4. TTie device of datm 3 wherein said viewer profile indudes information about the 
viewer derived from the monitoring means, said Information designed to predict tiie 

25 age, sex, family size, hobbies and interests of the viewer. 

5. The set-top device of daim 4 wherein the statistical correlation between viewer 
demographics and user viewing profile are used so that spedfic video sequences or 
indida are anonymously targeted to spedfic viewer demographics via specific viewer 
profile requ^ linked to the transmission of tiie said spedfic video sequences or 

30 indida. 

6. The set-top device of daim 5 wherein tiie Insertable indicia and the data pertaining to 
said insertable Indida are encoded into the vertical blanking interval, a spare audio 
channel, in the video itself, or in some other spare channel or field. 

7. The set-top device of daim 5 wherein a viewer can interactively amend, change, add 
35 to, or alter tiie viewer profile primarily for the purpose of solidting specific Indkda. 
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8. The set-top device of dalm 5 wherein the video signal can be a eompmssed video 
^nal that Is de«mp«ssed by sakl set-top device, or an analog or digital television 
broadcast. 

9. The set-top devtee of dalm 5 wherein said means for receiving Modes a c»-axia. 
5 able a,nnectlon, television aerial connecHon, satellite b^adcast connection, or 

telephone connection. 

10. The device Of dalmswherem the television can be substituted withaa,mp.ter or any 
Other video reception device. 

a. The set-top device Of dalm 10 wherein said Indlda indude links to Internet web-sltes 
12. A method of anonymous targeted Insertion of selected Indlda or sequences Into video 

broadcasts comprising the steps of: 
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a. 



monitoring the usage and viewing habits of a viewer of a television set or 
other Video recep«on device via a set-top device located at the television set or other 
video reception device; 

15 b. creating a local viewer profile In said device derived from data acquired 

from said monitoring step, said viewer profile Indicating certain *aracteristics of the 
Viewer; 

c. employing said locally stored viewer pmfile to dedde which Insertable 
video mdlda or sequence to Insert based upon the required viewer profile encoded in said 

20 video sequence; 

d. linking specific Insertable indlda matching spedfie viewer profiles; 

e. encoding Insertable Indlda as well as date perteinlng to the placement, 
shape, size, and perspective of the Indlda direcHy Into the broadcast vWeo; 

f. having said set-top devtae decode the broadcast video and perform the 
25 insertion of the indlda. 

13. The method of dalm 12 further comprising the steps of: 

g. sampnng a statistically representative sub-set of viewers to create 
statistical correlations between viewer demographics and the set-top user's viewer profile. 

30 
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